Creating General-Purpose Corpora Using Automated Search Engine Queries

The Internet is a natural source of linguistic data, providing an abundance of texts of various types in a large number of languages. These texts are already in electronic form suitable for corpus studies, either as downloadable pages, or as a resource to be searched using search engines. On the other hand, large representative corpora of the size of the British National Corpus (BNC, Aston and Burnard 1998) exist for very few languages, because they are expensive to build. They are absent even for major world languages, such as Chinese or French. Many ad-hoc text collections are available, but they are restricted in either their size or the variety of text types. Typically they are produced on the basis of out of copyright fiction (such as Project Gutenberg)1 or newswire/newspaper texts that are available in large quantities and relatively easy to acquire from their publishers (e.g., the Reuters corpus for English (Rose et al. 2002), or the Gigaword corpora for Arabic, Chinese and English (Cieri and Liberman 2002). News corpora are useful for many applications, such as development of gazeteers, parsing and word sense disambiguation algorithms, yet they cannot replace corpora representative of general language, such as the BNC, as